Make hot reloadable by cosmo0920 · Pull Request #51 · chronosphereio/calyptia-plugin

cosmo0920 · 2023-10-03T08:33:12Z

This PR needs to work with fluent/fluent-bit#7997.

Bind pre_run callbacks to ensure only once execution at plugin lifecycles.
Bind pause/resume callbacks to synchronize ordinary lifecycles of input plugins.
Add handling for Fluent Bit V2 formatted events
pre_exit callbacks are not binded for now. However, the PR plugin_proxy: core: expose more callbacks for golang plugins fluent/fluent-bit#7997 provides such callbacks.

niedbalski · 2023-10-03T09:41:19Z

@nicolasparada can you take a look at this one please?

pwhelan · 2023-10-05T12:54:09Z

IWFM! I'll do another test once #52 lands as well.

cosmo0920 · 2023-10-06T10:15:51Z

IWFM! I'll do another test once #52 lands as well.

I've been suffering for fatal more stack g0 error. Did you use Golang 1.20 or 1.21?
I got the error with Golang 1.12.1...

Aligned lifecycles for channels and goroutines on hot-reloading * Added pre_run callbacks for Golang input and output plugins to ensure execute once before entering the running state * Binded resume/pause callbacks for Input Golang plugins * Added pre_exit callbacks for Golang input and output plugins to ensure aligning lifecycles for Golang's channels and goroutines Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>

Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>

cosmo0920 · 2023-10-06T11:13:43Z

I confirmed that we needn't use guard for absent channels with #52 mechanism. No need to change the API usages on derived plugins.

pwhelan · 2023-10-06T13:52:54Z

IWFM! I'll do another test once #52 lands as well.

I've been suffering for fatal more stack g0 error. Did you use Golang 1.20 or 1.21? I got the error with Golang 1.12.1...

I used go 1.20 per your instruction. From the materials you sent me it's clear 1.21 will not work.

cosmo0920 · 2023-10-06T14:00:26Z

Ok, I got it

niedbalski

LGTM

cosmo0920 · 2023-10-16T11:27:41Z

@pwhelan Any thoughts?

pwhelan

IWFM!

I tested it with the http_loader plugin and master fluent-bit and it seems to at least work as well as before.

./fluent-bit-master -c crash-local.conf 
Fluent Bit v2.2.0
* Copyright (C) 2015-2023 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/10/20 12:30:39] [ info] [fluent bit] version=2.2.0, commit=bf17059bf6, pid=74476
[2023/10/20 12:30:39] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/10/20 12:30:39] [ info] [cmetrics] version=0.6.4
[2023/10/20 12:30:39] [ info] [ctraces ] version=0.3.1
[2023/10/20 12:30:39] [ info] [input:http_loader:http_loader.0] initializing
[2023/10/20 12:30:39] [ info] [input:http_loader:http_loader.0] storage_strategy='memory' (memory only)
[2023/10/20 12:30:39] [debug] Using storage key "http_loader_v2_678d5cf4"
[2023/10/20 12:30:39] [debug] Data loaded from local disk storage with key "http_loader_v2_678d5cf4"
[2023/10/20 12:30:39] [debug] Successfully json unmarshalled loaded data https://ggy66547.live.dynatrace.com/api/v2/auditlogs?nextPageKey=vu8y3hPZ3q0AAAAAAloaRWk33_UAAAGKlw7-6AAAAYqYWJXoAAAACgBCZXZlbnRUeXBlKExPR0lOLExPR09VVCxDUkVBVEUsREVMRVRFLEdFVCxQT1NULEdFTkVSQUwsUkVBRCxVUERBVEUpAb7vMt4T2d6t
[2023/10/20 12:30:39] [debug] Failed to json unmarshal stored request body, returning original string ""
[2023/10/20 12:30:39] [debug] Successfully processed loaded data with index index 686
[2023/10/20 12:30:39] [debug] [input:http_loader:http_loader.0] [thread init] initialization OK
[2023/10/20 12:30:39] [ info] [input:http_loader:http_loader.0] thread instance initialized
[2023/10/20 12:30:39] [ info] [output:stdout:stdout.0] worker #0 started
[2023/10/20 12:30:39] [ info] [output:syslog:devo_coud] setup done for localhost:5140 (TLS=on)
[2023/10/20 12:30:39] [ info] [sp] stream processor started
[2023/10/20 12:30:39] [debug] Collect invoked
[2023/10/20 12:30:39] [debug] Sending request: method="GET" url="https://ggy66547.live.dynatrace.com/api/v2/auditlogs?nextPageKey=vu8y3hPZ3q0AAAAAAloaRWk33_8AAAGKlw7-6AAAAYqYWJXoAAAACgBCZXZlbnRUeXBlKExPR0lOLExPR09VVCxDUkVBVEUsREVMRVRFLEdFVCxQT1NULEdFTkVSQUwsUkVBRCxVUERBVEUpAb7vMt4T2d6t" headers=map[Authorization:[Api-Token dt0c01.UGC5MJC2ZXMKX35HL3KRHH4H.VREOSCZVE36BVZ3LKOESI33X44GICB34K43EEJE7E2Y5DQOX4UFOBXAJSVNMIGYI] User-Agent:[Fluent-Bit HTTP Loader Plugin]] body=""
[2023/10/20 12:30:40] [debug] Received response: status_code=401 headers=map[Cache-Control:[no-store, no-cache] Content-Type:[application/json;charset=utf-8] Date:[Fri, 20 Oct 2023 15:30:40 GMT] Pragma:[no-cache] Server:[ruxit server] Server-Timing:[dtRpid;desc="-1092255344", dtSInfo;desc="0"] Set-Cookie:[dtCookie=v_4_srv_12_sn_569FA794E2D7E168621531932430BCF0_perc_100000_ol_0_mul_1_app-3Af6b10dd0df01cfe1_1_rcs-3Acss_0; Path=/; Domain=.dynatrace.com; secure] Strict-Transport-Security:[max-age=31536000;includeSubDomains] Vary:[Accept-Encoding] X-Oneagent-Js-Injection:[true] X-Robots-Tag:[noindex]] body="{\"error\":{\"code\":401,\"message\":\"Token Authentication failed\"}}"
[2023/10/20 12:30:40] [error] Error collecting data: template: pull_interval:1:208: executing "pull_interval" at <$fromtime.Add>: can't evaluate field Add in type string
2023/10/20 12:30:41 http_loader: next URL now: 2023-10-20 15:30:41.597126241 +0000 UTC
2023/10/20 12:30:41 http_loader: next URL totime: 2023-10-20T15:30:41
[2023/10/20 12:30:41] [error] Error collecting data: template: url:1:511: executing "url" at <$fromtime.Format>: can't evaluate field Format in type string
2023/10/20 12:30:42 http_loader: next URL now: 2023-10-20 15:30:42.600503841 +0000 UTC
2023/10/20 12:30:42 http_loader: next URL totime: 2023-10-20T15:30:42
2023/10/20 12:30:42 http_loader: next URL fromtime: 2023-10-20T15:30:41
2023/10/20 12:30:42 http_loader: next URL from: 2023-10-20T15:30:41, to: 2023-10-20T15:30:42
[2023/10/20 12:30:42] [debug] Sending request: method="GET" url="https://ggy66547.live.dynatrace.com/api/v2/auditlogs?filter=eventType%28LOGIN,LOGOUT,CREATE,DELETE,GET,POST,GENERAL,READ,UPDATE%29&from=2023-10-20T15:30:41&to=2023-10-20T15:30:42&pageSize=1&sort=timestamp" headers=map[Authorization:[Api-Token dt0c01.UGC5MJC2ZXMKX35HL3KRHH4H.VREOSCZVE36BVZ3LKOESI33X44GICB34K43EEJE7E2Y5DQOX4UFOBXAJSVNMIGYI] User-Agent:[Fluent-Bit HTTP Loader Plugin]] body=""
[2023/10/20 12:30:42] [debug] Received response: status_code=401 headers=map[Cache-Control:[no-store, no-cache] Content-Type:[application/json;charset=utf-8] Date:[Fri, 20 Oct 2023 15:30:42 GMT] Pragma:[no-cache] Server:[ruxit server] Server-Timing:[dtRpid;desc="1046871211", dtSInfo;desc="0"] Set-Cookie:[dtCookie=v_4_srv_12_sn_94474B2F387D93BA3DD84A63CC24D392_perc_100000_ol_0_mul_1_app-3Af6b10dd0df01cfe1_1_rcs-3Acss_0; Path=/; Domain=.dynatrace.com; secure] Strict-Transport-Security:[max-age=31536000;includeSubDomains] Vary:[Accept-Encoding] X-Oneagent-Js-Injection:[true] X-Robots-Tag:[noindex]] body="{\"error\":{\"code\":401,\"message\":\"Token Authentication failed\"}}"
[2023/10/20 12:30:42] [debug] Initiating store process with key "http_loader_v2_678d5cf4"
[2023/10/20 12:30:42] [debug] Successfully stored data in local disk storage with key "http_loader_v2_678d5cf4"
[2023/10/20 12:30:42] [ warn] Error updating metadata in cloud API with key "http_loader_v2_678d5cf4": cloud api: client not initialized or pipelineID missing
^C[2023/10/20 12:30:43] [engine] caught signal (SIGINT)
[2023/10/20 12:30:43] [ warn] [engine] service will shutdown in max 5 seconds
[2023/10/20 12:30:43] [debug] [input:http_loader:http_loader.0] thread pause instance
2023/10/20 12:30:43 goroutine will be stopping: name="http_loader"
[2023/10/20 12:30:44] [ info] [engine] service has stopped (0 pending tasks)
[2023/10/20 12:30:44] [debug] [input:http_loader:http_loader.0] thread pause instance
[2023/10/20 12:30:44] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2023/10/20 12:30:44] [ info] [output:stdout:stdout.0] thread worker #0 stopped
[2023/10/20 12:30:44] [debug] [input:http_loader:http_loader.0] thread exit instance

I also seem to be able to successfully reload the instance as well:

[2023/10/20 12:32:37] [engine] caught signal (SIGHUP)
[2023/10/20 12:32:37] [ info] reloading instance pid=74803 tid=0x7ffff6dffac0
[2023/10/20 12:32:37] [ info] [reload] stop everything of the old context
[2023/10/20 12:32:37] [ warn] [engine] service will shutdown when all remaining tasks are flushed
[2023/10/20 12:32:37] [debug] [input:http_loader:http_loader.0] thread pause instance
2023/10/20 12:32:37 goroutine will be stopping: name="http_loader"
[2023/10/20 12:32:38] [ info] [engine] service has stopped (0 pending tasks)
[2023/10/20 12:32:38] [debug] [input:http_loader:http_loader.0] thread pause instance
[2023/10/20 12:32:38] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2023/10/20 12:32:38] [ info] [output:stdout:stdout.0] thread worker #0 stopped
[2023/10/20 12:32:38] [debug] [input:http_loader:http_loader.0] thread exit instance
[2023/10/20 12:32:38] [ info] [reload] start everything
[2023/10/20 12:32:38] [ info] [fluent bit] version=2.2.0, commit=bf17059bf6, pid=74803
[2023/10/20 12:32:38] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/10/20 12:32:38] [ info] [cmetrics] version=0.6.4
[2023/10/20 12:32:38] [ info] [ctraces ] version=0.3.1
[2023/10/20 12:32:38] [ info] [input:http_loader:http_loader.0] initializing
[2023/10/20 12:32:38] [ info] [input:http_loader:http_loader.0] storage_strategy='memory' (memory only)
[2023/10/20 12:32:38] [debug] Using storage key "http_loader_v2_678d5cf4"
[2023/10/20 12:32:38] [debug] Data loaded from local disk storage with key "http_loader_v2_678d5cf4"

Big stamp of approval. This was with go 1.21 BTW.

cshared.go

niedbalski requested review from nicolasparada and niedbalski October 3, 2023 08:51

cosmo0920 force-pushed the cosmo0920-make-hot-reloadable branch 4 times, most recently from 6d38086 to d093473 Compare October 6, 2023 09:44

cosmo0920 force-pushed the cosmo0920-make-hot-reloadable branch 3 times, most recently from 9ee1106 to 2456a9d Compare October 6, 2023 10:27

cosmo0920 added 7 commits October 6, 2023 20:11

out_stdout: Migrate to use calyptia/plugin API

9a462e6

Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>

cshared: Handle event format of Fluent Bit V2

4df6533

Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>

cshared: test: Fix failing cshared tests

2f9fc7b

Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>

cshared: Make goroutine for Collect to be able to exit

24e9ea0

Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>

cshared: Make backward compatible and interruptable goroutines

81c0547

Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>

examples: out_gstdout: Update dependencies

4463036

Signed-off-by: Hiroshi Hatake <hatake@calyptia.com>

cosmo0920 force-pushed the cosmo0920-make-hot-reloadable branch from 2456a9d to 4463036 Compare October 6, 2023 11:12

pwhelan mentioned this pull request Oct 6, 2023

[FEAT] buffered channel drain single collect #52

Merged

cosmo0920 mentioned this pull request Oct 10, 2023

Enable execution for multi instances #54

Merged

niedbalski approved these changes Oct 10, 2023

View reviewed changes

niedbalski requested a review from pwhelan October 10, 2023 20:58

niedbalski self-assigned this Oct 19, 2023

pwhelan approved these changes Oct 20, 2023

View reviewed changes

cshared.go Show resolved Hide resolved

pwhelan merged commit aa19413 into main Oct 23, 2023

nicolasparada deleted the cosmo0920-make-hot-reloadable branch June 6, 2024 14:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make hot reloadable #51

Make hot reloadable #51
pwhelan merged 7 commits intomainfrom
cosmo0920-make-hot-reloadable

cosmo0920 commented Oct 3, 2023

Uh oh!

niedbalski commented Oct 3, 2023

Uh oh!

pwhelan commented Oct 5, 2023

Uh oh!

cosmo0920 commented Oct 6, 2023

Uh oh!

cosmo0920 commented Oct 6, 2023

Uh oh!

pwhelan commented Oct 6, 2023 •

edited

Loading

Uh oh!

cosmo0920 commented Oct 6, 2023

Uh oh!

niedbalski left a comment

Uh oh!

cosmo0920 commented Oct 16, 2023

Uh oh!

pwhelan left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

cosmo0920 commented Oct 3, 2023

Uh oh!

niedbalski commented Oct 3, 2023

Uh oh!

pwhelan commented Oct 5, 2023

Uh oh!

cosmo0920 commented Oct 6, 2023

Uh oh!

cosmo0920 commented Oct 6, 2023

Uh oh!

pwhelan commented Oct 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cosmo0920 commented Oct 6, 2023

Uh oh!

niedbalski left a comment

Choose a reason for hiding this comment

Uh oh!

cosmo0920 commented Oct 16, 2023

Uh oh!

pwhelan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pwhelan commented Oct 6, 2023 •

edited

Loading